Solving Visual Madlibs with Multiple Cues

摘要

This paper presents an approach for answering fill-in-the-blank multiplechoice questions from the Visual Madlibs dataset. Instead of generic andcommonly used representations trained on the ImageNet classification task, ourapproach employs a combination of networks trained for specialized tasks suchas scene recognition, person activity classification, and attribute prediction.We also present a method for localizing phrases from candidate answers in orderto provide spatial support for feature extraction. We map each of thesefeatures, together with candidate answers, to a joint embedding space throughnormalized canonical correlation analysis (nCCA). Finally, we solve anoptimization problem to learn to combine scores from nCCA models trained onmultiple cues to select the best answer. Extensive experimental results show asignificant improvement over the previous state of the art and confirm thatanswering questions from a wide range of types benefits from examining avariety of image cues and carefully choosing the spatial support for featureextraction.